N90-2G 


o o 


9 


Autonomous Sensor- Based Dual- Arm Satellite Grappling 


Brian Wilcox 


Kani I so Todd Lilwin Sanad Ilayati 


Jot Propulsion Laboratory 
California Institute of Techno ogy 
Pasadena. CA 91 109 


Bruce Bon 


Abstract 

The NASA Telerobot ic Research Project is exploring the feasibility of using robots in space 
for 011 -orbit, assembly, maintenance, and repair operations. Dual-arm satellite grappling is one 
of its more challenging tasks. 

The task involves the integration of technologies developed in the Sensing and Perception 
(SfcP) Subsystem for object acquisition and tracking, and the Manipulator Control and Mech- 
anization (MCM) Subsystem for dual-arm control. S.CP acquires and tracks the position, ori- 
entation, velocity, and angular velocity of a slowly spinning sitellite, and sends tracking data 
to the MCM subsystem. MCM grapples the satellite and brings it to rest, controlling the arms 
so that no excessive forces or torques are exerted on the satellite or arms. 

The demonstration setup includes a 350-pound satellite niockup which can spin freely on 
a gimbal for several minutes, closely simulating the dynamics of a real satellite. The satellite 
mockup is fitted with a panel under which may be mounted various elements such as line 
replacement modules and electrical connectors that will he used to demonstrate servicing tasks 
once the satellite is clocked. 

The subsystems are housed in three Micro VAX II microcomputers, 'flu' hardware of the SAP 
Subsystem includes CCD cameras, video digitizers, frame boilers, IMF EX (a custom pipelined 
video processor), a time-code generator with millisecond precision, and a MicroYAX 11 computer. 
Its software is written in Pascal and is based on a locally written vision software library. 1 lie 
hardware of the MCM Subsystem includes PUMA 5(50 robot, arms, Lord force/torque sensors, 
two Micro VAX II computers, and Unimat ion pneumatic parnlh I grippers. Its software is written 
in C, and is based oil a robot language called RCCL. 

This paper describes the two subsystems and provides test, results on the grappling of the 
satellite mockup with rotational rates of up to 2 rpm. 


1 Introduction 

X AS A is exploring the possibilities of using autonomous svst one. in place ol ast ronauts for dangerous 
or time-consuming operations, such as extra- vehicular activiy. The NASA lelerobol Research 
Project is looking at the use of autonomous systems for performing on-orbit assembly, maintenance, 
and repair operations. All of these operations present challenges to the field of robotics. 

'Phis paper is concerned with one challenge associated with the repair of artificial earth-orbiting 
satellites. That challenge centers around the fact that most such satellites are spin-stabilized. In 
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Figure 1: Testbed Setup 


order to repair such a satellite, it must first be de-spun. An investigation was made into the use of 
a dual-arm robot with visual assist to grapple and de-spin a rotating satellite rnockup. 

The Testbed of the NASA Telerobot Project, housed at the Jet Propulsion Laboratory, is 
composed of several subsystems. Two of these subsystems, the Sensing and Perception (S&P) 
Subsystem and the Manipulators and Control Mechanization (MCM) Subsystem, were directly 
involved in the effort to grapple a spinning satellite. S&P visually acquired and tracked the rotating 
satellite and transmitted the satellite’s state information to MCM in real time. MCM, using the 
data from S&P, directed the arms to grab the satellite and to stop its motion as shown in Figure 1. 

Below we will discuss these two subsystems and provide the details of the satellite grappling 
technique. 
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2 Testbed Setup 

The Testbed of the NASA Telerobot Project is a facility which is composed of many parts. It is 
divided into three main sections: the computer facility, the operator control station, and the test 

area. 

In the computer facility there are various computer systems to support the subsystems. Included 
among them are several MicroVAX II microcomputers. Sun workstations, and a Symbolics Lisp 
machine. There are also other specialized processors used at low levels of support. 

The operator control station provides a place for the operators during Testbed use. It has 
working surfaces, computer terminals, video monitors, and joysticks. It is situated so as to give a 
good view of the test area. 

The test area contains everything else. It houses the two manipulation arms, plus a third arm 
used for holding movable video cameras (not used for satellite grappling), and all of the video 
cameras. The satellite mockup, as well as other targets for t.elerobotic research, are in this area. In 
addition, there is a calibration fixture used to calibrate the cameras in S&P and the arms in MCM 
to a common coordinate system. 

Satellite Mockup 

The satellite mockup is a 350-pound model of a generic artificial satellite. It is suspended from a 
counterweight by a long cable which is connected — through a fairly wide opening in the top of 
the satellite — to a gimbal near its center of mass. The method of suspension allows the satellite 
to move within a small area in a manner similar to its free-space counterparts. 

The sides of the mockup consist of real solar panels, framed in the gold-foil insulating material 
typical of real satellites. This gives S&P a realistic visual target, complete with visual clutter and 
specular reflections. 

On one side of the mockup there is a removable task panel, under which may be mounted 
various elements such as line replacement modules and electrical connectors. These can be used to 
demonstrate servicing tasks once the satellite is grappled and docked. 

S&P Hardware 

The S&P Subsystem hardware consists of a DEC MicroVAX II microcomputer and several other 
pieces of special equipment. There is a time-code generator, shared with MCM, which allows 
time tagging the data to millisecond precision. There are five video cameras. Three of them are 
mounted to the rear of the test area, yielding good overall views of the work region. The other 
two are mounted on a robot arm and are used for close-up views; they were not used in the work 
described here. 

The cameras feed their signals into video digitizers, which in turn feed a custom video pipelined 
processor. The processor is called IMFEX, the Image Feature Extractor, and it uses a thresholded 
modified- Sobel operator to find high-contrast edges in video images. The output of IMFEX is fed 
into video buffers for storage, access, and manipulation by the MicroVAX II. The video buffers 
have internal D/A converters to allow viewing of their contents, which may be a captured frame, 
graphics generated by the MicroVAX II, or a combination of the two. A 16-by-16 video crossbar 
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switch is used to route all of the analog video signals between the components mentioned above, 
and over to the video monitors in the operator control station for display. 

S&P Software 

The S&P software is composed of two major sections, both written in VAX Pascal. The first section 
is a 30,000-line general-purpose vision software support library. It contains the software used to 
control and access the various hardware components of S&P, as well as routines to perform all 
manner of support services for machine vision. Included in this library are the routines used to 
perform the functions of acquisition and tracking described below, to perform camera calibration, 
and to deal with object models. The second major section of the software is a 20,000-line software 
package which is the S&P-specific application code. It is decomposed into five separate tasks, all 
running concurrently. 

MCM Hardware 

The MCM Subsystem hardware consists of two PUMA 560 robot arms. Each arm is equipped with 
a Unimation pneumatic parallel gripper which has been fitted with simple parallel fingers holding 
a special grappling tool. Each tool has a flexible handle and a Velcro pad at the end in order to 
hold to the satellite mockup, which has the two complementary Velcro pads attached on either 
side of the task panel. Each arm is also equipped with a commercial (Lord Corporation) wrist 
force/torque sensor. The wrist force/torque sensors are used to read the forces and torques sensed 
in all six dimensions. The computing hardware consists of two DEC MicroVAX II computers. 
Each one is interfaced to the LSI 11/73 microprocessors of the Unimate controller via two DRV11 
parallel cards. The Unimate controller reads the LORD force/torque sensor data through another 
DRV11 parallel card. The S&P object state data is obtained through a 9600 baud RS-232 serial 
line connected between the S&P MicroVAX to the MCM Micro VAXs. Figure 2 shows a schematic 
drawing of the hardware. 

MCM Software 

The MCM Micro VAXs run under a modified Berkeley BSD 4.3 Unix operating system. The main 
robot language is RCCL (Robot Control C Library) which was developed originally at Purdue 
University by Vincent Hayward [3] based on Richard Paul’s robotics textbook [4] and later enhanced 
by John Lloyd [5] and ported to a MicroVAX II [6]. RCCL is a collection of C functions which 
provide easy and general robot programming. The software can be partitioned into two main parts: 
the planning level and the real-time control level. 

The planning level consists of functions that allow the programmer to specify desired coordinate 
frames for the robot end-point target position. The programmer must specify several frames, such 
as where the robot is in the world frame and in the tool frame. He/she must also specify the 
velocity, and whether the motion should be carried out in Cartesian or joint-interpolated modes. 
The relationships between the various frames are entered in the code exactly as one would write 
them in mathematical notations. A function called Makeposition interprets the code and sets up 
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Figure 2: Schematic drawing of the hardware 


appropriate matrix equations. Its general form is: 

p = Makeposition (“P”, Z, . . . ,T6, . . . , J2, EQ,A, B , . . . , i7 t TOOL, R) 

This simply sets up a matrix equation 

(Z...)T6(...fl) = AB...U (1) 

The main objective is to solve this equation for T6 as 

T6 = (Z . . .y\AB . . . U ){. . . R)~ r (2) 

where T6 represents a homogeneous transformation relating the link 6 frame in the Oth (or shoulder) 
frame. The planning level runs in a normal time-shared manner. A program can develop many 
motions and place them in a queue for the control level to execute sequentially. An important 
feature of RCCL is that it allows one to modify the matrices in equation (1) in the control level. 
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These modifications can be triggered either by the planning level or by external sources such as 
from S&P object state data or force/torque data. This feature has been used extensively in our 
work to implement both the tracking and force/compliant control during grappling the satellite 
mockup. 

The control level which is called RCI (Robot Control Interface) is a general robot programming 
environment. One can write his/her own robot control programs and create custom trajectories to 
run the robots. In the normal RCCL operations this level receives the motions from the motion 
queue and uses a trajectory generator to interpolate between the specified via points. Finally, 
inverse kinematics is performed to obtain the joint angle set points to be sent to the joint micro- 
processors via the LSI 11/73 computer. 

The system works by a hardware clock (located in the Unimation controller) which periodically 
sends an interrupt to a communication program in the LSI 11/73. At every interrupt, this program 
gathers information from the arm — which includes the joint angles and other sensors that have 
been interfaced to the LSI 11/73 — and makes them available to the Micro VAX in shared memory; it 
signals the Micro VAX with a hardware interrupt. The Micro VAX reads these data and immediately 
sends the joint angles that have been computed in the previous cycle back to the LSI 11/73 for 
execution. Note that in this implementation the VAL language is completely bypassed. The 
hardware of the Unimate controller remains intact, however, and one can switch between the VAL 
language and RCCL without any hardware modifications. 

3 S&P Acquisition and Tracking Algorithms 

The task of watching a moving object is broken down into two stages, the first of which is called 
acquisition. This is the stage wherein the object of interest is first localized within the views of the 
cameras, and an initial computation is made as to its location and movement within 3-space. It is 
computationally intensive, and cannot perform quickly enough on currently available computers to 
keep up with a moving object in real time. 

The second stage is call tracking. Tracking is more computationally efficient than acquisition, 
and is used to follow the object as it moves, updating the state information that was initially 
provided by acquisition. 

Both stages compute the following time-tagged state information in three dimensions: position, 
translational velocity, orientation, angular velocity, and a covariance matrix of these values. 

Acquisition 

A fully autonomous acquisition algorithm is currently under development at the Jet Propulsion 
Laboratory and was not tested in the grappling experiment. A moment’s reflection, however, will 
reveal that it is not possible to track an object without first acquiring it in some fashion. In order to 
satisfy this need, a quick-and-dirty operator- assisted version called hand acquisition was designed 
for the current work. 

In hand acquisition, an a priori position is used as a starting point. Using this position, S&P 
displays a wire-frame projection of the satellite’s object model in the display, superimposed on the 
raw video. While the satellite was held still, the operator uses the joysticks to move the wire-frame 
overlay — and thus the state of the object model — until it roughly overlaps the satellite mockup 


312 



in all camera views. Once complete, the operator signals that tracking may start. Hand acquisition 
of a moving satellite was attempted with mixed results. 

Tracking 

The tracking algorithm was designed by Donald B. Gennery. Detailed descriptions of the algorithm 
are given elsewhere [1,2]. The tracker performs its operations in five major phases. In the first 
phase it acquires a frame of video and notes the time tag associated with the data. In the second 
phase the old object state is propagated forward to the time of the new data. In the third phase a 
projection of the propagated object state’s edges is made into the view of the camera which took the 
new data. In the fourth stage measurements are made of the discrepancies between the locations 
of edge points in the projected edges and the locations of edge points in the data. In the fifth and 
final stage the projected object state is adjusted by using a least-squares technique with respect to 
the measurements taken in the fourth stage. Uncertainties are propagated and determine the effect 
that any given data set has on the current object state. 

These five stages constitute one tracking cycle. Between cycles, the updated state information 
is sent to MCM. Then the tracker selects the next camera and performs another tracking cycle. It 
continues in this fashion until told to stop or until it loses track. If track is lost, S&P cycles back 
into acquisition and repeats the entire process. 

4 MCM Tracking and Grappling Algorithms 

The two robots are driven independently by two MicroVAXs. They run the same copy of the 
software except each has its own coordinate frames because of the different locations of the robots 
and because they grapple different points on the satellite mockup. Two machines are used because 
the computing power of one MicroVAX is not adequate to control two robots during the tracking 
phase. The two robots are coordinated because they are basically driven from the same source of 
data — the S&P object states of the satellite mockup. 

The MCM software receives the satellite object state at a rate of about two times per second. 
The object state consists of a time-stamp, position vector and orientation quaternion of the centroid 
of the satellite mockup, the translational and angular velocities, and their covariances. 

A complete cycle of satellite grappling is comprised of four phases: approach, tracking, grappling, 
and docking. 

In the approach phase, the MCM software monitors the orientation and angular velocity of the 
satellite mockup. It deploys the robots to the approach positions when the satellite mockup is 
spinning with a rate at or below two rpm. The approach positions are chosen so that the robots 
have the maximum work space for tracking and grappling. The approach position of the left robot 
is described by the follow equation: 

qT L ° 6 T L 6 e T L = a Tjf, (3) 

where %T L is a transformation describing the Oth frame, *T L is the tool frame, and ™T L is the 
approach frame of the left robot as shown in Figure 3. 

The robots wait in the approach positions until the mockup has rotated such that the pads are 
facing the tools with a designed distance of 100mm. At that moment the robots start tracking the 
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Figure 3: Coordinate Frame Assignments 

satellite mockup driven by the S&P object state data. At the same time the distances between the 
tools and the pads are gradually reduced until they contact each other. The following kinematic 
equation is used to specify the motion in the tracking phase: 

o t l ° 6 t l %t l = *r ;t l ( 4 ) 

where is a transformation describing the centroid of the satellite mockup frame, and ‘T L is the 
transformation from the satellite mockup centroid to the left pad. The distance between the tools 
and the pads, initially 100mm, is faked in equation (4) by making the tool 100mm longer than its 
physical length. This distance is reduced during tracking until contact. 

In RCCL, one can generate trajectories by using the trajectory generator or by an external 
means. The latter is made possible since one can modify any of the transformations except T6 
in equation (1) in real time. The satellite tracking uses the latter strategy since it would be too 
time-consuming if planning is done each time MCM receives an updated object state. 

Hence tracking is done in the control level which drives the robots in the following way: Every 
time an object state is received, the current frame of the satellite is predicted according to the 
received data, and the frame of the tool is computed from the joint angles of the robot. The 
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difference between p Ti and is computed from 

a l = ;t l = ' w t l ;t l 

where W T = X p T. In order to track the satellite, the robots are required to have moved 
this A by the time the next state update is received. This means that, in every sampling period, 
the robots are moved by <5 = A X j>, where t is the robot control sampling period, and T is the 
inter-arrival period of the object state. 

The tools approach the grappling pads until a contact is initiated. This is sensed by the 
force/torque sensors and after the contact the motion is changed from vision-based servoing to 
force servoing. The following kinematic equation is used to specify the motion in the grappling 
phase: 

oTl %T l \Tl = w a T s p T L COMPLY (6) 

where COMPLY is a “small” time- varying transformation and has the following form 


COMPLY = 
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( 7 ) 


Since the COMPLY transformation is placed after the s p T L , it will modify the ideal trajectory by 
a small amount each sample time. Because of integral force control, the COMPLY transformation 
will be modified based on the force/torque sensor readings and will keep its value even after the 
forces have been nulled. Since the satellite mockup cannot be stopped instantaneously once it is 
grappled, the software decelerates the mockup according to a trajectory which is generated based 
on the initial velocity at the moment of contact. 

Once the satellite mockup is stopped, it is pulled to the docking fixture. Active force control 
is used to nullify the force built up due the dual-arm coordinated motion. If grappling does not 
occur — detected by the lack of contact between the tools and pads — the whole cycle is repeated 
by letting the satellite mockup spin one more time. 


5 Conclusions and Future Improvements 

We have successfully grappled the satellite mockup with rotational rates of up to 2 rpm. With 
higher speed, due to the communication delay and control inaccuracies, the robots start to miss 
the pads. 

In the present MCM implementation, the computation is performed with two MicroVAX com- 
puters, which limits its control rate to once every 28 msec. In the near future, we plan to port our 
software to a Sun 4/260 computer which will increase the control rate to 200 Hz. This increase 
will improve the force control capability and hence reduce the build-up of forces at the moment of 
contact and subsequent grappling. The improved version of RCCL [7,8] can coordinate trajectories 
for two robots. As such, more precise coordination both at the planning and control levels can be 
achieved. 
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